It passed the following intra-node tests and all inter-node tests:

torchrun --nnodes=1 --nproc_per_node=2 benchmarks/benchmark_uccl.py --ipc

torchrun --nproc_per_node=8 benchmarks/benchmark_uccl_collective.py --ring

torchrun --nproc_per_node=2 tests/test_engine_nvlink.py

The basic idea is to use conn->is_local() to check uds_sockfd_, which is only used by intra-node communication. For inter-node communication, it is set to -1.

The one-sided operations (write/read) require different metadata types depending on the communication: FifoItem for RDMA and IpcTransferInfo for IPC. So, use opaque-metadata overloads (void const* meta, size_t meta_len) to let pybind pass raw bytes from advertise() without knowing the underlying type. The C++ side checks conn->is_local() to deserialize as IpcTransferInfo or FifoItem accordingly.

Thanks @zhongjiechen, this is great. I see the changes don't apply for readv/writev since there is no readv_ipc/writev_ipc. Would you be adding that?

Thanks @zhongjiechen, this is great. I see the changes don't apply for readv/writev since there is no readv_ipc/writev_ipc. Would you be adding that?

I think we can support that in another PR.

Hi @zhongjiechen , we recently upgrade P2P from UDS to shm-backed jring in #718. So you might need to resolve some conflict